Thinking Hallucination for Video Captioning

AbstractWith the advent of rich visual representations and pre-trained language models, video captioning has seen continuous improvement over time. Despite performance improvement, models are prone to hallucination. Hallucination refers generation highly pathological descriptions that detached from source material. In captioning, there two kinds hallucination: object action Instead endeavoring learn better a video, in this work, we investigate fundamental sources hallucination problem. We identify three main factors: (i) inadequate features extracted (ii) improper influences target contexts during multi-modal fusion, (iii) exposure bias training strategy. To alleviate these problems, propose robust solutions: (a) introduction auxiliary heads trained multi-label settings on top (b) addition context gates, which dynamically select fusion. The standard evaluation metrics for measures similarity with ground truth captions do not adequately capture relevance. end, new metric, COAHA (caption assessment), assesses degree Our method achieves state-of-the-art MSR-Video Text (MSR-VTT) Microsoft Research Video Description Corpus (MSVD) datasets, especially by massive margin CIDEr score.